Simulating three-dimensional views using planes of content

ABSTRACT

Approaches enable image content (e.g., still or video content) to be displayed in such a way that the image content will appear, to a viewer, to include portions with different locations in physical space, with the relative positioning of those portions being determined at least in part upon a current relative position and/or orientation of the viewer with respect to the device, as well as changes in that relative position and/or orientation. For example, image content can be grouped or otherwise contained or assigned to different planes, levels, or other such groupings of content. The planes of content can enable image content included within those planes to be displayed to provide a viewer with an appearance or view of the content that appears to be positioned and/or displayed in 3D space. As that viewing angle changes, the content can be re-rendered or otherwise updated to display the image content from a perspective that reflects the change in viewing angle.

BACKGROUND

As the capabilities of various computing devices increase, and as peopleare utilizing computing devices for an increasing variety of tasks, theexpectations of users of these devices continues to increaseaccordingly. As an example, an increasing number of applications areattempting to meet these expectations by providing a virtual reality,enhanced reality, or three-dimensional experience. While some devicesutilize three-dimensional displays that require specific hardware, suchas special viewing glasses, these can be expensive and complex, and canprovide varying levels of user satisfaction. A large number of devicesstill utilize conventional two-dimensional displays or provide contentthat is substantially created in two dimensions. While certain shadingor rendering can be utilized to give the impression of three-dimensionalcontent, the content will typically not act like a truethree-dimensional object or scene, as changes in position, orientation,or lighting will generally not be updated realistically in the display.Thus, the virtual nature of the displayed content can be significantlydegraded.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an example situation where a user can view contentand interact with a computing device in accordance with variousembodiments;

FIG. 2 illustrates an example state of an interface that can be renderedin accordance with an embodiment;

FIG. 3 illustrates an example state of an interface that can be renderedin accordance with various embodiments;

FIGS. 4( a) and 4(b) illustrate various states of an interface that canbe rendered in accordance with various embodiments;

FIGS. 5( a) and 5(b) illustrate various states of an interface that canbe rendered in accordance with various embodiments;

FIG. 6 illustrates an example state of map information that can begenerated in accordance with various embodiments;

FIGS. 7( a) and 7(b) illustrate various states of the map informationthat can be rendered in accordance with various alternate embodiments;

FIGS. 8( a), 8(b), and 8(c) illustrate various states of levels ofgraphical icons that can move by level in accordance with variousembodiments;

FIGS. 9( a) and 9(b) illustrate various states of book content inaccordance with various embodiments;

FIG. 10 illustrates an example process for updating a display ofinformation to account for orientation changes in accordance withvarious embodiments;

FIG. 11 illustrates an example process for determining a relativeposition of a viewer that can be used in accordance with variousembodiments;

FIG. 12 illustrates an example device that can be used to implementaspects of the various embodiments;

FIG. 13 illustrates example components of a client device such as thatillustrated in

FIG. 11;

FIGS. 14( a)-14(f) illustrate example approaches to determining headposition and/or gaze direction that can be used in accordance withvarious embodiments;

FIGS. 15( a) and 15(b) illustrate example approaches to determiningchanges in the relative distance to a user in accordance with variousembodiments

FIGS. 16( a)-16(d) illustrate example approaches to determining changesin the relative viewing angle for a user in accordance with variousembodiments;

FIGS. 17( a) and 17(b) illustrate an example approach to determining therelative position of a user that can be utilized in accordance withvarious embodiments

FIGS. 18( a) and 18(b) illustrate an example approach to determiningdevice motion that can be utilized in accordance with variousembodiments; and

FIG. 19 illustrates an environment in which various embodiments can beimplemented.

DETAILED DESCRIPTION

Systems and methods in accordance with various embodiments of thepresent disclosure may overcome one or more of the aforementioned andother deficiencies experienced in conventional approaches to displayingcontent using an electronic device. In particular, various embodimentsenable image content (e.g., still or video content) to be displayed insuch a way that the image content will appear, to a viewer, to includeportions with different locations in physical space, with the relativepositioning of those portions being determined at least in part upon acurrent relative position and/or orientation of the viewer with respectto the device, as well as changes in that relative position and/ororientation. The content can include various portions, and differentadjustments can be applied to each portion based upon these and/or othersuch changes. These adjustments can include, for example, changes due toparallax or occlusion, which when added to the rendered content inresponse to relative movement between a viewer and a device can enhancethe experience of the viewer and increase realism for content renderedon a two- or three-dimensional display screen.

For example, in at least some embodiments, image content can be groupedor otherwise contained or assigned to different planes, levels, or othersuch groupings of content. These groupings may be specified throughsoftware, the user, etc. In some embodiments, the groupings may beprovided via the operating system on the device, where differentportions of an image or other content to be displayed can be assigned todifferent layers, levels, etc. The planes of content can enable imagecontent (e.g., images, text, etc.) included within those planes to bedisplayed to provide a viewer with an appearance or view of the contentthat appears to be positioned and/or displayed in 3D space. For example,some of the planes (and the content included in those planes) can appearcloser to (or above) a surface of a display screen of a device (andhence the viewer), while other planes of content (and the contentincluded in those planes) “fall back” or appear smaller in 3D space,appearing to be further from the surface of the display screen.

The planes of content can be associated with position information (e.g.,a lateral position and depth position) and the position information canbe used by the device to render (i.e., generate data for the display ofthe planes of content) a view of the content based on the relativeposition, direction, and/or orientation between the viewer and device toprovide a two- or three-dimensional representation of that content thatis appropriate for that viewing angle, giving the impression of athree-dimensional view or display even when the display is in twodimensions. Further, the position information can be used to rendershadows based on an intersection of light from a virtual light sourcewith one of the planes of content.

As the relative position of the viewer and/or orientation of the devicechanges, the position information for corresponding planes of contentcan be updated, and the updated position information can be used toadjust the perspective from which the planes of content are rendered tocorrespond to changes in the relative viewing angle of the viewer.Further, shadows and shading can be adjusted to be appropriate for theperspective. For example, as the viewer tilts, rotates, or otherwisechanges the orientation of the device, or as the viewer's relativeposition or orientation changes with respect to the device, the planesof content can appear to translate laterally, move back and forth inapparent distance from the surface of the screen, or otherwise changeshape or appearance. The relative movements can be based upon factorssuch as the distance of the viewer to the device, a direction ofmovement of the user, a direction of change in orientation of thedevice, or other such factors. The relative movements can be selectedsuch that the different planes of content appear to be positioned inthree dimensions with respect to each other, and act appropriately withchanges in relative position and/or orientation, and thus viewing angle,of the viewer. Additionally, the planes of content might be lighted orshaded according to one or more light sources near the device, and thelighting direction can be updated in response to movement of the deviceto shade the objects in the image according to the position of the lightsource.

In various embodiments, the relative position and/or orientation of aviewer of a computing device can be determined using at least one imagecapture element of the device. For example, the feed from a video cameracan be analyzed to locate a relative position of the viewer in the videofeed, which can be analyzed to determine the relative direction of theviewer. In other embodiments, one or more digital still cameras cancapture images periodically, in response to detected movement of theviewer and/or device, or at other appropriate times, which then can beanalyzed to attempt to determine viewer position, as distance can oftenbe determined in addition to direction when analyzing multiple sourcesof information from different locations. Distance can be determined, forexample, using stereoscopic imaging or proximity sensing, among othersuch options. In some embodiments, infrared (IR) imaging can be used todetect specific features of the viewer, such as the viewer's eyes, foruse in determining and/or tracking the location of the viewer. In stillother embodiments, changes in the orientation and/or position of thedevice can be determined using at least one motion sensor of the device,in order to provide for a higher sampling frequency than might otherwisebe possible using the image information captured by the camera, orotherwise attempt to improve the relative position determinations. Insome situations, a sensor that is remote, separate, or otherwise incommunication with the device can be used to detect a change inorientation and/or position of the device. The orientation informationcan be received at the device from the sensor, and the device can causethe appearance of the interface to be altered based at least in part onthe received orientation and/or position information. Accordingly, aviewer can view and interact with the planes of content, and canmaneuver through the planes of content using various approachesdiscussed herein.

Based at least in part upon the determined direction of the viewer, thedevice can determine a primary viewing angle with respect to the planeof the display screen, and thus the planes of content (e.g., a scene) tobe rendered and displayed on the device. For at least certain types ofcontent, the device can adjust the rendering to provide a two- orthree-dimensional representation of that content that is appropriate forthat viewing angle, giving the impression of a three-dimensional view ordisplay even when the display is in two dimensions. For example, aschanges in the relative position, direction, and/or orientation betweenthe viewer and device are determined, a set of transformation equationsand/or coefficients for those equations to adjust a scale and atranslation for the content can be determined. The transformationequations can be used to adjust the perspective from which the planes ofcontent are rendered to correspond to changes in the relative viewingangle of the viewer. In this way, the equations can be used to determinehow to display or otherwise alter the appearance of the planes ofcontent in 3D space (e.g., such as by bringing an element “forward”toward the front of the display screen or bringing an element “back”from the front of the display screen), laterally, etc. For example, inaccordance with an embodiment, the transformation equations are used toapply a scale effect that mimics that which would accompany aperspective frustum by scaling about a fixed pivot point in the centerof the display screen. As the position of the viewer to the devicechanges, the simulated frustum is skewed to the side by translatingdeeper planes of content laterally based on their depth. For example, alateral motion of a user with respect to a computing device can have anassociated change in angular direction with respect to a normal of adisplay screen of the device. Using basic geometry, the change in anglecan result in different lateral translations of objects in differentplanes of content, based at least in part upon the virtual distancebetween those planes. Planes and objects included in those planes thatare intended to appear closer to the user will then be translated by agreater amount than planes and objects in those planes that are intendedto appear further from the user, in order to provide the impression ofobjects positioned in three-dimensional space.

In at least some embodiments, a computing device can attempt todetermine changes in the relative position, direction, and/ororientation between the viewer and device in order to update theperspective from which the displayed content is rendered or otherwisedisplayed. For example, the device can continue capturing and analyzingimage information to attempt to determine changes in relative positionof the viewer, such as may be based on movement of the viewer and/or thedevice. The device also can utilize information from at least oneorientation or position determining element of the device, such as anaccelerometer or inertial sensor, to assist in detecting motions of thedevice and updating the viewing angle accordingly. These elements alsocan detect changes in orientation of the device, such as throughrotation of the device, even though the relative position between theviewer and the device might not have substantially changed. The displaycan be updated based at least in part upon changes in orientation aswell. By adjusting the perspective from which the image content isrendered to correspond to changes in the relative viewing angle of theuser, a three-dimensional representation can be generated on a two- orthree-dimensional display screen that is consistent, across multipleviewing angles, with actual or virtual three-dimensional content.

The ability to update a perspective of rendered content can provideadditional advantages as well. For example, an object included in theplanes of content might at least partially obscure or occlude anotherobject. Using conventional displays, a viewer would not be able to viewthe occluded content. By enabling the rendering perspective to changebased upon relative position or orientation, a viewer can effectivelylook “around” the occlusion to view the content that was previously notvisible in the display. Further, the amount by which the occlusion movesupon a change in position or orientation can be indicative of a relativeheight or distance between the occlusion and the other content, whichcan be useful for mapping or other such applications.

Various other applications, processes, and uses are presented below withrespect to the various embodiments.

FIG. 1 illustrates an example situation 100 wherein a user 102 isinteracting with a computing device 104. Although a portable computingdevice (e.g., a smart phone, an electronic book reader, or tabletcomputer) is shown, it should be understood that various other types ofelectronic device that are capable of determining and processing inputcan be used in accordance with various embodiments discussed herein.These devices can include, for example, notebook computers, personaldata assistants, video gaming consoles or controllers, portable mediaplayers, and wearable computers (e.g., smart watches, smart glasses,etc.) among others. In this example, the computing device 104 includes acamera 106 positioned on a side or corner of the device such that theimaging element will likely be able to capture image information of atleast a portion of the user while the user is viewing content displayedon the device. For example, the imaging element 106 in FIG. 1 is on thefront of the device such that an angular capture range 108 of theimaging element can image at least a portion of the user while theviewer is viewing content displayed on the display element of theelectronic device. In accordance with various embodiments, being able tocapture image information for the user enables the device to determine arelative position and/or orientation of the user with respect to thedevice and adjust a display of content on the device in accordance withthat position and/or orientation.

For example, the display screen can present the appearance of 3D, or3D-like behavior, but might be a standard 2D display. Accordingly, in atleast some embodiments, image content (e.g., images, text, etc.)corresponding to content on different planes, levels, or other suchgroupings of content can enable the image content included within thoseplanes to be displayed to provide a viewer with an appearance or view ofthe content that appears to be positioned and/or displayed in 3D space.In this way, various embodiments enable image content (e.g., images,text, advertisements, etc.) included on these planes to appear in 3D,such as by bringing a plane (and the content included on the plane)“forward” or otherwise causing the plane to appear to be positionedtowards the front of the display screen (or above) in a 3D display orquasi-three-dimensional rendering on a 2D display element of the device,while other planes (and the content on those planes) appear to bepositioned “backwards” or at a greater distance from the front of thedisplay screen. Further, the rendering can utilize 3D mappinginformation, such as a set of layer depths or z-levels, to determine howto relate various interface planes to each other.

FIG. 2 illustrates an example 200 of an interface displayed on a displayscreen 202 of a computing device 200. In this example, a user is viewinga conventional 2D representation of a webpage 201. As with manyconventional webpages, the areas of the webpage can be divided intozones or areas depending on the content type, the markup of the webpage,visual classification of the webpage, and/or white space analysis of thewebpage. In this example, portions of image content (e.g., planes orlayers of content, or other interface elements) can include a header204, article text 206, at least one image 208, at least one link 210,advertisements 214, and various other links 216. It should beunderstood, however, that aspects of the various embodiments can be usedwith a variety of types of interface, which can include a wide varietyof different portions of image content.

In a conventional 2D representation, the portions of image content, suchas the interface elements described, can be organized in a number ofdifferent ways. One such way is to organize the interface elements in ahierarchy, such as one that includes parent and child nodes. Aninterface element can be associated with a node, such as a parent nodeand one or more other interface elements can be associated with childnodes. The nodes can be part of a node hierarchy, as may be provided viathe operating system and/or other software on, or remote from, thecomputing device. The relationship between the nodes (e.g., parent-childrelationship in a node hierarchy) can include position information andthe position information can be used to determine the relative positionof one interface element to another interface element. Accordingly,using the hierarchy, such as any node hierarchy of interface elements,developers can quickly and efficiently modify the appearance and/oractions of the interface elements to create an interactive interface.However, because the positioning information includes relativepositioning information (i.e., the relative position of one interfaceelement to another interface element), and does not include depthinformation (e.g., a position depth or screen-space positioninformation, it becomes difficult to render a layout or view as a 3Dscene with parallax, shadows, and 3D perspective without the absoluteposition of the interface elements relative to the display screen.

Accordingly, in accordance with various embodiments, image content(e.g., images, text, etc.) can be grouped or otherwise contained orassigned to different planes, levels, or other such groupings of contentwhile working within a conventional 2D framework by injecting containercontrols with appropriate attributes into layout files. The planes ofcontent can enable the image content included within those planes to bedisplayed to provide a viewer with an appearance or view of the contentthat appears to be positioned and/or displayed in 3D space. For example,planes of content can be arranged be associated with a depth position(e.g., a z-position), where the depth position can be measured relativeto the display screen or relative to a depth position of another planeof content. The depth position of the planes of content can be used todisplay the planes at different depths and/or display shadows based onthe depth and lateral position of the planes of content. The depthposition may be specified through software, the user, a developer ofcontent, etc. For example, the device can specify a default depthposition for the content in the situation where no depth information hasbeen provided. In this way, the planes have the effect of manipulatingthe z-position (i.e., the appearance of depth relative to a displayscreen of the computing device) of the plane and the image contentincluded in the plane. The planes of content can be associated with asubset of a plurality of nodes of a node hierarchy, where each node caninclude screen-space position data instead of, or addition to, relativeposition data. Thus, each node can include the screen-space positiondata for an associated plane, where the screen-space position data caninclude position information for a planes screen-space position. Inaccordance with various embodiments, a screen-space position can includethe position of plane relative to the display screen, where the positioncan include a lateral position (x/y coordinates) and a depth position(z-position) of the plane relative to the display screen. Based on thescreen-space position of different planes of content, various depthrelationships (i.e., the depth of one plane of content to another planeof content) can be determined).

The screen-space coordinates for a respective node can be updated basedon changes in that nodes corresponding plane's position due toanimations or scrolling, and/or changes in orientation of the device oruser of the user relative to the device. The tracking can occurautomatically. Accordingly, the positioning data provided by this systemcan enable several important effects. For example, the relativepositions of two views at different depths that do not share a directparent-child relationship allow shadows to be cast correctly betweenthem. It also gives the necessary information to compute perspectiveeffects, as it is necessary to know where each plane is with respect tothe shape of a view frustum. Additionally, if true 3D content isembedded into the views, the position information allows the 3D contentto be oriented correctly based on the 3D content's location in 3D space.

For example, FIG. 3 shows an example 320 of an interface 301 displayedon a display screen 302 of a computing device 300. As shown, theinterface includes a number of interface planes or levels of contentarranged at various depths. Image content can be grouped or otherwisecontained or assigned to the interface planes. The image content caninclude, for example, a header 304, article text 306, at least one image308, at least one link 310, advertisements 314, and various other links316. The interface planes can be displayed at different depths (orz-levels) on the display screen 302 of computing device 300. Theseinterface planes may be specified through software, the user, adeveloper of content, etc. In some embodiments, the interface planes maybe provided via the operating system on the device, where differentportions of an image or other content to be displayed can be assigned todifferent planes, levels, etc. The interface planes can enable imagecontent (e.g., images, text, etc.) included within those planes to bedisplayed to provide a viewer with an appearance or view of the contentthat appears to be positioned and/or displayed in 3D space. In someembodiments, an interface plane whose position along the z-axis can bedescribed in relation to either its parent of the display screen.

In this example, image content other links 316 is displayed at thehighest level while advertisements 314 are displayed at the lowestlevel. Accordingly, each plane (and content included on the plane) canbe rendered to correspond to a particular depth, such as may bedescribed in relation to either a planes parent plane or the displayscreen. In some situations, an interface plane can be inside (i.e.,nested) another interface plane. For example, image 308 and link 310 areon a first plane that is inside a second plane that includes articletext 306. In this situation, the second plane that includes the articletext 306 is a parent interface plane to the first plane that includesimage 308 and link 310. As such, the first plane that includes image 308and link 310 can inherent the depth of its parent plane (i.e., thesecond plane) and can be offset by some relative amount from its parentplane. Further, in accordance with various embodiments, by usinginterface planes to specify positioning along the z-axis, all descendentviews may benefit from one or more visual cues to create a sense ofdepth. These effects can include shadow casting, parallax, and depthscaling, among others.

For example, FIG. 4( a) shows an example state 401 of an interface whereimage content is grouped or otherwise contained or assigned to differentplanes, levels, or other such groupings of content. As descried, thesegroupings may be specified through software, the user, etc and can bearranged at various depths. In some embodiments, the groupings may beprovided via the operating system on the device, where differentportions of an image or other content to be displayed can be assigned todifferent layers, levels, etc. As shown in FIG. 4( a), there are threelevels of planes arranged according to three different depths, z₀, z₁,and z₂. On the first level (i.e., the lowest depth z₀ or the depthappearing to be furthest away from a user and/or the display screen) isinterface plane 400. On the second level (i.e., z₁) is interface plane402 and interface plane 404. In this example, interface plane 402 andinterface plane 406 are child interface planes of parent interface plane400. On the third level are interface planes 410, 412, and 414.Interface planes 410 and 412 are child planes to interface plane 402 andgrandchildren planes to interface plane 400. Interface plane 414 is achild plane to interface plane 404 and a grandchild plane to interfaceplane 400. Thus, interface planes 400, 402, 404, 410, 412, and 414 canbe represented in a node hierarchy with interface plane 400 as a root ofthe hierarchy, interface planes 402 and 404 as branches of the root, andinterface planes 410, 412, and 414 as leaves of the hierarchy, withinterface planes 410 and 412 branching from interface plane 402 andinterface plane 414 branching from interface plane 404. As described,the interface planes can be associated with nodes of a hierarchy. Asdescribed, the interface planes can be organized in a node hierarchy,where at least a portion of the nodes correspond to interface planes andother nodes correspond to content included in those planes. At the rootof the node hierarchy there can there may be a root interface plane(e.g., interface plane 400). Within the root interface plane, there maybe at least one 2D layout container. Examples of 2D layouts can includeframe layouts, linear layouts, relative positioning layouts, grid-basedlayouts, among others. Two-dimensional layout containers can include oneor more child 2D layout containers that can include content such as oneor more user interface elements. Two-dimensional child layout containerscan include one or more of their own child layout containers which canalso include content such as one or more user interface elements. Inother embodiments, a 3D layout container provide for additionalcapabilities such as the appearance of depth, scale, and translationeffects. It will be appreciated by those of ordinary skill in the artthat a user interface could have fewer or greater depths and/or fewer orgreater UI elements than are illustrated in FIG. 4( a). Thus, thedepiction of the user interface 401 in FIG. 4( a) should be taken asbeing illustrative in nature and not limiting to the scope of thedisclosure.

As described, the interface planes can be displayed at different depths(or z-levels) on a display screen of computing device and theseinterface planes may be specified through software, the user, adeveloper of content, etc. In some embodiments, the interface planes maybe provided via the operating system on the device, where differentportions of an image or other content to be displayed can be assigned todifferent planes, levels, etc. The interface planes can enable imagecontent (e.g., images, text, etc., 409, 411, 413, 415) included withinthose planes to be displayed to provide a viewer with an appearance orview of the content that appears to be positioned and/or displayed in 3Dspace. In some embodiments, an interface plane whose position along thez-axis can be described in relation to either its parent of the displayscreen. For example, some of the planes (and the content included inthose planes) can appear closer to (or above) a surface of a displayscreen of a device (and hence the viewer), while other planes of content(and the content included in those planes) “fall back” or appear smallerin 3D space, appearing to be further from the surface of the displayscreen.

In accordance with various embodiments, the planes of content can beassociated with position information (e.g., a lateral position and depthposition such as x, y, z coordinates) and the position information canbe used by the device to render a view of the content based on therelative position, direction, and/or orientation between the viewer anddevice to provide a two- or three-dimensional representation of thatcontent that is appropriate for that viewing angle, giving theimpression of a three-dimensional view or display even when the displayis in two dimensions. Further, the position information can be used torender shadows based on an intersection of light from a virtual lightsource with one of the planes of content. In accordance with variousembodiments, a virtual light source can be positioned within the modelused by the renderer from which light appears to come from when contentis rendered. The virtual light source can appear to be positioned in anynumber of positions where the light from the light source can be used torender shadows. In some situations the virtual light source can have afixed position or can have a position that attempts to mimic real lightsources. Example positions can include behind the viewer of thecomputing device, from one of the corners of the computing device, amongothers.

For example, FIG. 4( b) shows an example state 460 of an interface wherebased on a change in the relative position, direction, and/ororientation between a viewer and the device, the display of theinterface planes (and content included in those planes) is updated.

For example, upon detecting the change in the relative position,direction, and/or orientation between the viewer and the device, thenode hierarchy can be redrawn or rendered to correspond to the change.In this example, interface planes 400, 402, 404, 410, 412, and 414 caneach be transformed according to a rotation, scale, translation,perspective projection, among other possibilities, based on the changeso as to give the appearance that the interface planes exist in a 3Denvironment. As described, interface planes have the effect ofmanipulating the z-position (i.e., depth position relative to thedisplay screen) of it and its content. In various embodiments, this canbe expressed visually through scale and parallax effects. Parallax isthe situation where if the relative position, direction, and/ororientation between a viewer and the device is changed, the interfaceplanes and their content can be automatically translated by anappropriate amount to create a parallax effect. The translation effectexhibited by a given interface plane can be proportional to its distancefrom the display screen. As the distance increases, interface planes canbe translated by a greater amount. Interface planes with a zero depthare right up against the display screen and therefore may not exhibitparallax effects. In the situation of depth scaling, interface planesand their contents can be scaled to appear larger if they are closer tothe camera. Interface planes and their content that are deeper into thebackground can be rendered smaller and can be offset based on therelative position, direction, and/or orientation between the viewer andthe device to create a compelling depth illusion as the orientationchanges.

An example of this is shown in FIG. 4( b), where interface plane 414 iscapable of partially obscuring interface plane 402 as well as interfaceplane 400 located a lower depth. As further shown, as the relativeposition of the viewer and/or orientation of the device is changed fromthe position in FIG. 4( a) to the position in FIG. 4( b), the positioninformation for corresponding planes of content can be updated, and theupdated position information can used to adjust the perspective fromwhich the planes of content is rendered to correspond to changes in therelative viewing angle of the viewer. For example, as the viewer tilts,rotates, or otherwise changes the orientation of the device, or as theviewer's relative position or orientation changes with respect to thedevice, the planes of content can appear to translate laterally, moveback and forth in apparent distance from the surface of the screen, orotherwise change shape or appearance. The relative movements can bebased upon factors such as the distance of the viewer to the device, adirection of movement of the user, a direction of change in orientationof the device, or other such factors. The relative movements can beselected such that the different planes of content appear to bepositioned in three dimensions with respect to each other, and actappropriately with changes in relative position and/or orientation, andthus viewing angle, of the viewer. In accordance with variousembodiments, content included in an interface plane can inherent depthrelated effects (e.g., translation and scaling effects). For example,content 409 which lies flat against interface plane 404 can inherent anychange in translation or scale do to its relationship (i.e., beingincluded in the interface plane) with interface plane 404. Similarly,content 411 can inherent any change in translation and/or scale do toits relationship with interface plane 410 and content 413 can inherentany change in translation and/or scale do to its relationship withinterface plane 412. As described, content can include images, text,appearance effects such as highlighting, color, shading, etc.

As described, the position information can be used to render shadowsbased on an intersection of light from a virtual light source with oneof the planes of content. For example, FIG. 5( a) shows an example state501 of an interface that includes three levels of planes. On the firstlevel is interface plane 500. On the second level is interface plane 506and interface plane 508. Interface plane 506 and interface plane 508 arechild interface planes of parent interface plane 500. As shown,interface plane 506 can cast a shadow 507 on interface plane 500, andinterface plane 504 can cast a shadow 505 on interface plane 500. On thethird level is interface planes 510, 512, and 514. Interface planes 510and 512 are child planes to interface plane 506 and cast respectiveshadows 560 and 570 on interface plane 506. Interface plane 514 is achild plane to interface plane 504 and casts shadow 580 on interfaceplane 504.

As the relative position of the viewer and/or orientation of the devicechanges, the position information for corresponding planes of content isupdated, and the updated position information is used to adjust theperspective from which the planes of content is rendered to correspondto changes in the relative viewing angle of the viewer. Further, shadowsand shading can be adjusted to be appropriate for the perspective. Forexample, as the viewer tilts, rotates, or otherwise changes theorientation of the device, or as the viewer's relative position ororientation changes with respect to the device, the planes of contentcan appear to translate laterally, move back and forth in apparentdistance from the surface of the screen, or otherwise change shape orappearance. The relative movements can be based upon factors such as thedistance of the viewer to the device, a direction of movement of theuser, a direction of change in orientation of the device, or other suchfactors. The relative movements can be selected such that the differentplanes of content appear to be positioned in three dimensions withrespect to each other, and act appropriately with changes in relativeposition and/or orientation, and thus viewing angle, of the viewer.Additionally, the planes of content might be lighted or shaded accordingto one or more light sources near the device, and the lighting directioncan be updated in response to movement of the device to shade theobjects in the image according to the position of the light source.

For example, FIG. 5( b) shows an example state 560 of an interface,using the known relative location of a virtual light source and thechange in orientation, the device can also determine the properdirection to the light source in the new orientation, and as such cangenerate shadows for the interface planes elements based at least inpart upon the lighting direction, as would be consistent for the currentuser viewing angle. The ability to adjust shadowing with the changes indisplay in a realistic way can help to enhance the user experience, andcan also help the user to better understand the direction and/or textureof, for example, a two-dimensional representation of a three-dimensionalelement. As shown, the perspective from which the planes of content arerendered is updated, as well as the perspective of shadows cast fromthose planes. In this example, as the perspective of interface elements506, 510, 512, and 514 is updated, their respective shadows 507, 560,570, and 580 are updated.

In a conventional layout, the draw or view of the layout of theinterface is in a fixed order. However, in the situation where viewscontain interface planes with content at different depths, the desireddraw order becomes an important factor to ensure correct timing torender shadows. In accordance with various embodiments, draw order isthe order in which images and other objects are displayed on a displayscreen of a computing device. In some situations, a default order or anorder set by a developer of the interface. In this example. In thisexample, the draw order of interface planes can begin with the interfaceplane at the lowest level. In this example, plane 500 is drawn firstbecause it is at level 1. In accordance with various embodiments, whenan interface plane's draw code path is executed, that plane sets astatic flag indicating that it is currently drawing. Next, shadowscasted on that interface plane are drawn before drawing any planes atthe second level. In this example, shadows 507 and 505 are drawn beforeinterface planes 504 and 506 are drawn. Any descendant that is at thesame depth is drawn. The descendant planes can be drawn in a defaultorder or an order specified by a developer. In this example, descendantor child interface planes 504 and 506 are drawn. If while drawinginterface planes 504 and 506, any of planes 510, 512, 514 begin to draw,these interface planes will first check the state of its ancestorinterface planes (e.g., interface planes 504 and 506). If an ancestorplane is drawing, the interface plane attempting to draw (e.g., one ofinterface planes 510, 512, or 514) will stop drawing and add itselfalong with any state needed to a priority queue based on depth to deferits rendering until a later time. Once all descendants have either drawnor deferred, a current interface plane may render any shadows being caston it and then may set its flag to indicate it is ready to draw a newlevel and begin to iterate through the deferred interface planes indepth order. For example, upon drawing interface planes 504 and 506,shadows 560, 570 and 580 are drawn on the respective interface plane.Finally, interface planes 510, 512, and 514 are drawn to complete theprocess.

FIG. 6 illustrates an example device 600 displaying map content on adisplay element 602 of the device. In this example, the user has enteredan address into a mapping application, and mapping information isgenerated for display on the display element 602, including a pin ormarker 610 indicating the approximate location of the address on the mapregion and pin or marker 611 indicating the user's home. In thisexample, the pin or markers 610 and 611 are rendered at a depth abovethe map. However, when the user (not shown) is in a default position (orwithin a default range of positions) with respect to the device, such assubstantially in front of the display screen, the pins or markers appearrelatively flat. It should be noted that approaches to locating anaddress or location and generating map information are well known in theart and, as such, will not be discussed herein in detail.

As discussed, it can be desirable in at least certain embodiments toenhance the realism of such a situation as much as possible. One way isto add shading to the image such that the pin and buildings displayedappear to be three-dimensional objects. Generally, the shadows arerendered from a fixed direction and applied to a particular view, suchas a top-down view as illustrated. If the user moves the device, ormoves relative to the device, however, the shading will not change andthe perspective of the device will not adjust to show buildings or thepin from appropriate views resulting from the rotation, which can takethe viewer out of the experience. Similarly, the appearance of the itemswill not adjust if the user moves relative to the device, such that theuser will be aware that the display is a two-dimensional rendering.

Systems and methods in accordance with various embodiments can takeadvantage of any of a number of elements that can be used to determinechanges in relative position and/or orientation between a user and anelectronic device. For example, the device 600 in FIG. 6 includes animaging element 604 which can be used to capture image information fordetermining a relative position or direction of a user as mentionedabove. An orientation-determining element 606, such as an accelerometer,electronic gyroscope, or inertial sensor, can determine changes in theposition or orientation of the device. Other input elements 608, such asmicrophones or proximity sensors, can be used as well in otherembodiments. The information from at least some of these elements can beanalyzed to determine a current viewing angle from the perspective ofthe user. By determining the current viewing angle, for example, thedevice can render content that corresponds substantially to athree-dimensional view of the content from the perspective of the user.

For example, FIG. 7( a) illustrates an example orientation of the device700 upon the device being rotated along a primary axis of the device,although a similar change in relative orientation could result frommotion of the user as will be discussed in more detail later herein. Ascan be seen, the rotation of the device triggered a corresponding changein the map information 704 displayed on the device. For example, insteadof seeing a top view of each building, the sides of various buildingsare displayed corresponding to the current viewing angle of the user.Thus, even though the display is a two-dimensional display, the renderedview of a building can be such that the display will present a view ofthat building that is similar to what the user would see if viewing athree-dimensional version of a building from the current viewing angle.In this example, where the left edge of the device is rotated out of theplane of the Figure, the left side of various buildings is rendered(along with a portion of the roof or other perspective-appropriateportions) based on the viewer direction being substantially orthogonalto the plane of the Figure.

In FIG. 7( a), the rendering of the location pins 702 and 711 have alsoupdated accordingly. In FIG. 4, the pins were shown in a substantiallytop-down view. In FIG. 7( a), the location pins 702 and 711 are renderedto appear to be above the map content and rendered for the currentviewing angle of the user. As described, portions of image content, suchas the location pins 702 and 711, can be organized in a hierarchy, suchas one that includes parent and child nodes. The location pins can beassociated with a node, where the nodes can be part of a node hierarchy.The relationship between nodes (e.g., parent-child relationship in anode hierarchy) can include position information and the positioninformation can be used to determine the relative position of oneinterface element to another interface element. The position informationcan also include screen-space position data that can include positioninformation for a pins screen-space position, such as a lateral position(x/y coordinates) and a depth position (z coordinate) of the interfaceelement relative to the display screen. In this example, the map contentcan be at first level (i.e., be associated with a first depth) and thepins can be at second level, where the second level is “taller” than thefirst level and content associated with the second level appears closerto the display screen. In addition to changing the way the pin isdisplayed, the user can now view information that might have previouslybeen hidden or occluded by the pin in the top view. For example, in FIG.4 the W in “Washington Street” was occluded by the location of the pin.In FIG. 7( a), the rotation of the device has resulted in the renderingof the pin changing to reflect the current viewing angle, which alsoresults in the W in Washington Street now being viewable by the user.Thus, the rendering changes not only the perspective of various elementsbut can also move those elements appropriately relative to any otherelements or occluded portions in order to further provide the sense of athree-dimensional world. A user thus can adjust the relative orientationbetween the user and the device to view information occluded by anobject, which may not have been possible in conventional approacheswithout manually moving or removing the pin.

Since the user would not have been able to see the full name ofWashington Street in either of the previous orientations, the user cancontinue to adjust the relative orientation of the device until thedesired information is viewable. For example, in FIG. 7( b) the user cantilt the top of the screen forward out of the plane of the Figure,causing a different rendering of objects in the image. In this example,a side of the buildings towards the top of the device can be seen, and a“height” of the pin 702 is adjusted based upon the new orientation. Inthis example, the name of the street that was previously occluded nowcan be seen in the displayed image information, as “Washington Street”is now visible in the image. Further, the position information can beused to render shadows based on an intersection of light from a virtuallight source with one of the planes of content.

Another advantage of changing the orientation is that the user can alsoview different angles of occlusions that might not otherwise have beenobvious to the user. For example, in FIGS. 6 and 7 there is not muchdistinction in the display of Washington Street and Lincoln Avenue,other than their respective locations. In the rendering of FIG. 7( b),however, the orientation illustrates that Lincoln Avenue is in fact araised street 752, as the position of the street shifts upon orientationchange due to the street being at a different distance or plane.Further, shadowing 754 or other elements (e.g., posts or arches) can beadded to further illustrate the difference in location and perspective.Using conventional top-down views, the user might not have been able todiscern that Lincoln Street was actually above the other nearby streets,and could not be turned directly onto from either of the one way streetsthat cross under Lincoln Avenue.

An approach in accordance with various embodiments can instead utilizelayers of graphical elements that can move at different rates withrespect to each other, providing a sense of depth and three-dimensionalmovement. This can include rendering a view to have at least two (and inmany situations more) different “levels” or z-depths, where the upperlevel of some interface elements is rendered to appear near the outersurface of the display screen and the upper level of other interfaceelements can be rendered to appear at a lower level to the interface(e.g., separated a distance from the outer surface of the displayscreen).

For example, FIG. 8( a) illustrates a display 800 including three layersof graphical elements 802, 804, 806, such as graphical iconsrepresenting applications, folders, shortcuts, or any other type ofobject known or used on electronic devices to access various types offunctionality or information. In this example, a first layer of elements802 is rendered “over” a second layer of elements 804, which is renderedover a third layer of elements 806. It should be understood that therecan be any number of levels including any appropriate number ofelements, and that the ordering of the various layers can be adjusted orselected using any of a number of potential approaches, such as sortingor navigating the various layers. In this example, the elements of eachlayer are rendered with a different relative size, providing to the usera sense of distance of the various elements. In various embodiments, thedevice can use information such as the field of view of the camera, aswell as the position of the user's head or eyes to determine a currentpoint of view of a user, and the point of view can be used to render aninterface on a display screen or other such element of the computingdevice. The rendering can update as the determined point of view changesas a result of movement of the user and/or the computing device. Therendering can utilize 3D mapping information, such as a set of layerdepths or z-levels, to determine how to relate various interfaceelements to each other.

In order to enhance that sense of distance, as well as to provide asense of space and enable the user to obtain different views of thevarious elements, the layers can also move laterally with respect toeach other at different rates, where the rate of movement is coupledwith the relative size for that layer. For example, in FIG. 8( b) therehas been relative movement between the user and the device on which theimage information is displayed. As can be seen, the first layer ofelements 802 that is “closest” to the user has moved by the greatestamount. The second layer of elements 804 has moved by a smaller amount,representative of their respective distances, with the third layer ofelements 806 moving the least, if at all. In fact, in embodiments whereit is desired to keep the information substantially centered on thedisplay, the third layer of elements might actually move in the oppositedirection, as illustrated in FIG. 8( b), although the net relativemovement between could remain the same in either approach. As can beseen, in addition to providing a sense of three-dimensional space, theability to rotate the view enables different elements to be seen, whichcan help the user to locate and/or navigate to an element of interest,which might otherwise be hidden or occluded by an overlying element. Formap views, for example, each block of buildings might be assigned to adifferent layer, enabling the user to make distance determinations andmore accurately determine location information from thequasi-three-dimensional view.

In various embodiments, the device can adjust the appearance of shadowsassociated with the relevant interface elements to make those elementsappear to be higher in the interface, as well as to give a 3D appearanceas each shadow can move in position relative to an associated element asthe point of view changes, for example, to give the impression of anactual shadow being case by the relevant element. Further, the interfacecan render sidewalls or other elements that appear to provide a depth ofthe interface element from the point of view of the user, and the extentand shape of these elements can adjust with changes in point of view, aswell as an orientation of the device. Various other behaviors can beused as well to mimic 3D behavior as well as an appearance of stackedinterface elements. In at least some interfaces, there might be morethan three levels, and the amount of shadowing, color adjusting, andother such aspects can depend at least in part upon the level with whichthe element is associated.

In some embodiments, the graphical elements, as shown in display 820 ofFIG. 8( b) can be three-dimensional blocks that can be rotatedindividually as well, in order to show a least a side portion asdiscussed elsewhere herein. In this example, however, the elements areessentially “flat” or otherwise unable to rotate individually, with thethree-dimensional feel being generated primarily by the differences inlateral translation. While the inability of the individual elements torotate can potentially lessen the three-dimensional experience for someusers with respect to rotatable elements, the amount of processingcapacity can be significantly less and the rendering time less for atleast some devices, which can enhance the overall user experience. Asfurther shown in display 840 of FIG. 8( c), as the relative position,direction, and/or orientation between the viewer and device is changed,the rendering of the displayed content is updated. For example, in FIG.8( c), the view has been updated due to a change in the orientation ofthe device to show a side view of the elements 802, 804, and 806.

FIGS. 9( a) and 9(b) illustrate different views of an electronic book(e-book) 900, or similar content, that can be displayed on an electronicdevice using various approaches discussed herein. In FIG. 9( a), theuser (not shown) is in a default position (or within a default range ofpositions) with respect to the device, such as substantially in front ofthe display screen. Accordingly, the user can obtain a conventional viewof the text 902 in the e-book at that viewing angle. In this view, thelinked text 920 and 922 (i.e., text linking to other content) appearsflat and otherwise part of the other text. While reading the e-book, theuser might want to obtain certain information, such as how far the useris from the end of the book, how far it is until the next chapter, orexpose or otherwise emphasize text that links to other content and/or ishighlighted (e.g., underlined or bolded). Accordingly, a user can changea viewing angle of the user with respect to the device, such as byrotating the device or moving the user's head, to view a “side” of theelectronic book on the display. As illustrated in FIG. 9( b), the usercan tilt the device to see a view that includes a representation of theedges 904 of the pages of the book between the current page and the endof the book. Additionally, titling the device causes the links to appearto be above the text on the page, and as such, the user can quicklyidentify the links. Such an approach can provide value to the user, andalso potentially make the e-book experience more like reading an actualbook. Additional information can be added as well. For example, the edgeof the book can include not only an indicator 908 of the end (i.e., backcover) of the book, but can also include other indicators 906 to certainsections as well. For example, the side of the e-book can includeindicators marking the location of the next chapter and/or subsequentchapters, as well as the location of various notes, bookmarks,highlights, etc. Using such an approach, a user can tilt the device tosee how far until the end of the chapter, for example, to determinewhether to continue reading until the end of the chapter or end thecurrent viewing session at a different location. In some embodiments,the rendered image can also be manipulated (e.g., stretched or otherwisedeformed) in order to make the view of an object from the perspective ofthe user seem as if the display screen is a piece of glass through whichthe user is looking, rather than a conventional display screen in whichthings become increasingly compressed as the viewing angle increases.

FIG. 10 illustrates an example of a first portion 1000 of a process forproviding a relative orientation-based image display that can be used inaccordance with various embodiments. It should be understood that, forany process discussed herein, there can be additional, fewer, oralternative steps performed in similar or alternative orders, or inparallel, within the scope of the various embodiments unless otherwisestated. In this example, position tracking of a viewer is activated 1002on the device. In some embodiments a user must activate this modemanually, while in other modes the device can activate the modeautomatically when a person is detected nearby. Other modes ofactivation are possible as well, such as upon a user opening a specificapplication on the device. When the position tracking is active, thedevice can begin imaging 1004 around the device, whether in alldirections, some directions, a specific range of directions, or adirection substantially toward a determined viewer. As discussedelsewhere herein, in some embodiments the imaging will involve ambientlight image or video capture, while in other embodiments a device canutilize infrared imaging, heat signature detection, or any other suchapproach. The device can analyze 1006 the captured image information toattempt to locate features of a user, or at least a person nearby, wherethose features in some embodiments include at least the eyes, nose, orhead of a user. In some embodiments, the device will attempt to locatean object that is shaped like a human head and that contains twoeye-like features. In other embodiments, facial recognition or any othersuch algorithm can be used to attempt to determine the presence of ahuman head, or other portion or feature of a user, in the field of viewof at least one of the imaging elements.

Once the user features are located, the device can attempt to determine1008 aspects or information relating to those features. In this example,the determined aspects can be used to attempt to determine 1010 arelative orientation between the device and the user, as well as theorientation of those features relative to the device in at least someembodiments, which can be useful in determining information such as thepresent viewing location of a user. Image content can be displayed 1012based on the determined viewpoint. As described, image content (e.g.,still or video content) to be displayed in such a way that the imagecontent will appear, to a viewer, to include portions with differentlocations in physical space, with the relative positioning of thoseportions being determined at least in part upon a current relativeposition and/or orientation of the viewer with respect to the device, aswell as changes in that relative position and/or orientation. Thecontent can include various portions, and different adjustments can beapplied to each portion based upon these and/or other such changes. Forexample, in accordance with various embodiments, portions of imagecontent (e.g., planes or layers of content) can appear to be positionedand/or displayed in 3D space such that that some of the planes ofcontent appear closer to a surface of the display screen of the device(and hence the viewer), while other planes of content “fall back” orappear smaller in 3D space, appearing to be further from the surface ofthe display screen. The determined aspects then can be monitored 1014over time, such as by continuing to capture and analyze imageinformation to determine the relative position of the user and/ororientation of the device. In at least some embodiments, anorientation-determining element such as an accelerometer or electronicgyroscope can be used to assist in tracking the relative location of theuser and/or current relative orientation of the device. A change in theaspect, such as a change in position or orientation, can be determined1016, and the device can determine 1018 whether that change requires anadjustment to the image to be displayed. For example, an applicationmight require the device to be rotated a minimum amount before adjustingthe displayed image content, such as to account for a normal amount ofuser jitter or other such movement that may not be intended as input.Similarly, certain embodiments might not utilize continuous rotation,but might change views upon certain degrees of change in relativeorientation. If the orientation change is sufficient to warrant anadjustment, the device can determine and perform 1020 the appropriateadjustment to the content that is displayed in the image information,such as to adjust the screen-space position of interface elementsincluded in the image information.

For example, as described, the image can include portions of imagecontent (e.g., planes or layers of content). Each plane of content canbe an interface element, such as a shape, text, object, etc. Asdescribed, the interface elements can be organized in a hierarchy ofnodes that can include at least one parent node and one or more childnodes. The relationship between the nodes (e.g., parent-childrelationship in a node hierarchy) can include position information andthe position information can be used to determine the relative positionof one interface element to another interface element. Accordingly,using the hierarchy, such as any node hierarchy of interface elements,developers can quickly and efficiently adjust the appearance and/oractions of the interface elements to create an interactive interface.Further, each node can include screen-space position data. As described,screen-space position data can include the position of an interfaceelement relative to the display screen, where the position can include alateral position (x/y coordinates) and a depth position (z coordinate)of the interface element relative to the display screen. Thescreen-space coordinates for the nodes can be updated based on changesin a node's position due to animations or scrolling, and/or changes inorientation of the device or user of the user relative to the device,and the updated position information can be used to adjust a view of theimage information. Further, the position information can be used torender shadows based on an intersection of light from a virtual lightsource with one of the planes of content.

As an example of one such adjustment, FIG. 11 illustrates a secondportion 1100 of a process for modifying the image in response to adetermined change in orientation that can be used in accordance withvarious embodiments. During operation, an electronic device can acquire1102, using a camera of the device, at least one first image of a viewerof the device. The device can determine 1104 (and monitor over time), byanalyzing the first image, a first viewing direction, viewing angle, orlocation of the viewer with respect to the device. The device canassociate 1106 each one of a plurality of planes of content to acorresponding one of a subset of a plurality of nodes of a nodehierarchy, where each plane of the plurality of planes of contentincludes content capable of being displayed on a display screen of acomputing device. The device can determine 1108 depth relationshipsamong the plurality of planes of content based at least upon positioninformation included in the subset of the plurality of nodes, where theposition information includes a lateral position relative to the displayscreen and a depth position relative to the display screen for acorresponding one of the plurality of planes of content. In accordancewith various embodiments, depth relationships can include a depthposition (e.g., a z-position) of one plane of content relative to thedisplay screen or relative to a depth position of another plane ofcontent. These interface planes may be specified through software, theuser, a developer of content, etc. For example, the device can specify adefault depth position for the content in the situation where no depthinformation has been provided. Position information for a plurality ofplanes of content can be determined 1110, where the position informationfor each plane of the plurality of planes of content can include a firstlateral position and a first depth position relative to the displayscreen. As described, planes or layers of content, or other interfaceelements can include a header, article text, at least one image, atleast one link, advertisements, and various other links. It should beunderstood, however, that aspects of the various embodiments can be usedwith a variety of types of interface, which can include a wide varietyof different portions of image content.

An application executing on the device (or remote to the device) canutilize mapping, the position information, or other such data to renderimage content from a perspective associated with the first location ofthe user. For example, the device can display 1112, on the displayscreen, the plurality of planes of content, where each plane of theplurality of planes of content is displayed according to the respectivefirst lateral position and the respective first depth position, and thesubset of the plurality of planes of content further being displayedaccording to the respective first lateral offset.

Once the viewing direction of the user is determined, the device canattempt to monitor 1112 or detect changes in the viewing direction,relative position, or location of the user, as may result from changesin position or orientation of the user and/or the device. Changes in therelative position can be analyzed to determine whether the change isactionable 1114, such as where the change meets a minimum movementthreshold. In some embodiments, small movements might not result inadjustments in the display, in order to account for jitter or subtlevariations due to the user holding a device, for example, that are notintended as input to change the perspective. In various embodiments,there also must be a minimum amount of movement in order to justify there-rendering of the displayed image content. For example, cellularphones and portable media players might not have a significant amount ofgraphics processing capacity, such that continually attempting tore-render content using three-dimensional maps or other such informationcan consume a significant amount of resources of the device, slowingdown the rendering and potentially other functionality. Further,continually performing complex renderings can significantly drainbattery power for at least certain devices. Thus, for at least somedevices it can be desirable to at least wait to re-render the image froma different perspective until there is a minimum amount of movement,etc.

If there is no actionable movement, the device can continue to monitorthe relative position of the user. If there is actionable movementdetected, the device can attempt to determine the new relative position,orientation, and/or viewing angle, using any of the approaches discussedor suggested herein. For example, the device can determine 1116, byanalyzing at least one second image acquired by the camera, a secondviewing direction of the viewer with respect to the computing device.The device can then determine, for the subset of the plurality of planesof content, at least a second lateral offset corresponding to the secondviewing direction. The device can then update 1118 the display of theplurality of planes of content, each plane of the plurality of planes ofcontent being displayed according to the respective first lateralposition and the respective first depth position, the subset of theplurality of planes of content further being displayed according to therespective second lateral offset. As discussed, the image informationcan be manipulated (e.g., rotated, stretched, compressed, translated,etc.) to provide a consistent quasi-three-dimensional view as discussedelsewhere herein. As discussed, additional information can be added aswell, such as shadowing from a nearby light source. In at least someembodiments, an application can attempt to provide consistency in therendering and shading from any of a number of different viewing anglesconsistent with a three-dimensional display, even when the element usedto display the image information is two-dimensional in nature.

For example, as discussed, the initial relative position can bedetermined using the image information, and changes in that relativeposition can be determined using a combination of the image informationand the motion sensor information. For the current relative user pointof view, the interface can be rendered such that a 2D representation ofcontent is displayed on a display screen of the device. As described,the content can include one or more interface elements (e.g., planes ofcontent), which can include a header, article text, at least one image,at least one link, advertisements, and various other links. When achange in orientation of the device is detected, a three-dimensional ormulti-layer appearance, or other such aspect of at least a subset of theone or more interface elements can be rendered. This can includebringing an element “forward” in the rendering such that the elementappears to be close to the surface of the display screen in a 3D displayor quasi-three-dimensional rendering on a two-dimensional (2D) displayscreen. In accordance with various embodiments, bringing the elementforward can involve, for example, adjusting a size, shape, shadowing,focus/blur, and/or coloring of the element. For example, interfaceelements can appear to be positioned and/or displayed in 3D space suchthat that certain interface elements (e.g., text, images, etc.) becomelarger in 3D depth and/or appear closer to a surface of a display screenof the computing device, while other interface elements (e.g.,advertisements) “fall back” or appear smaller in 3D depth. As the usertilts, rotates, or otherwise changes the orientation of the device, theinterface elements can move back and forth or otherwise change shape orappearance. When the user views the display screen from a defaultposition (or within a default range of positions) with respect to thedevice, such as substantially in front of the display screen, aconventional view of the content can be displayed (e.g., the elementscan be moved “backwards” or otherwise be rendered in 2D, such as byperforming an opposite or alternative adjustment to that which wasperformed when the element was shown to be active and the renderingprocess for that interface can end). Further, in accordance with variousembodiments, at least one shadow can be rendered based at least in parton an intersection of light from a virtual light source with one of thesubset of the plurality of planes of content displayed based at leastupon the depth relationships among the plurality of planes of content.

FIG. 12 illustrates front and back views of an example electroniccomputing device 1200 that can be used in accordance with variousembodiments. Although a portable computing device (e.g., a smartphone,an electronic book reader, or tablet computer) is shown, it should beunderstood that any device capable of receiving and processing input canbe used in accordance with various embodiments discussed herein. Thedevices can include, for example, desktop computers, notebook computers,electronic book readers, personal data assistants, cellular phones,video gaming consoles or controllers, television set top boxes, andportable media players, among others.

In this example, the computing device 1200 has a display screen 1202(e.g., an LCD element) operable to display information or image contentto one or more users or viewers of the device. The display screen ofsome embodiments displays information to the viewers facing the displayscreen (e.g., on the same side of the computing device as the displayscreen). The computing device in this example can include one or moreimaging elements, in this example including two image capture elements1204 on the front of the device and at least one image capture element1210 on the back of the device. It should be understood, however, thatimage capture elements could also, or alternatively, be placed on thesides or corners of the device, and that there can be any appropriatenumber of capture elements of similar or different types. Each imagecapture element 1204 and 1210 may be, for example, a camera, acharge-coupled device (CCD), a motion detection sensor or an infraredsensor, or other image capturing technology.

As discussed, the device can use the images (e.g., still or video)captured from the imaging elements 1204 and 1210 to generate athree-dimensional simulation of the surrounding environment (e.g., avirtual reality of the surrounding environment for display on thedisplay element of the device). Further, the device can utilize outputsfrom at least one of the image capture elements 1204 and 1210 to assistin determining the location and/or orientation of a user and inrecognizing nearby persons, objects, or locations. For example, if theuser is holding the device, the captured image information can beanalyzed (e.g., using mapping information about a particular area) todetermine the approximate location and/or orientation of the user. Thecaptured image information may also be analyzed to recognize nearbypersons, objects, or locations (e.g., by matching parameters or elementsfrom the mapping information).

The computing device can also include at least one microphone or otheraudio capture elements capable of capturing audio data, such as wordsspoken by a user of the device, music being hummed by a person near thedevice, or audio being generated by a nearby speaker or other suchcomponent, although audio elements are not required in at least somedevices. In this example there are three microphones, one microphone1208 on the front side, one microphone 1212 on the back, and onemicrophone 1206 on or near a top or side of the device. In some devicesthere may be only one microphone, while in other devices there might beat least one microphone on each side and/or corner of the device, or inother appropriate locations.

The device 1200 in this example also includes one or more orientation-or position-determining elements 1218 operable to provide informationsuch as a position, direction, motion, or orientation of the device.These elements can include, for example, accelerometers, inertialsensors, electronic gyroscopes, and electronic compasses.

The example device also includes at least one communication mechanism1214, such as may include at least one wired or wireless componentoperable to communicate with one or more electronic devices. The devicealso includes a power system 1216, such as may include a batteryoperable to be recharged through conventional plug-in approaches, orthrough other approaches such as capacitive charging through proximitywith a power mat or other such device. Various other elements and/orcombinations are possible as well within the scope of variousembodiments.

FIG. 13 illustrates a set of basic components of an electronic computingdevice 1300 such as the device 1200 described with respect to FIG. 12.In this example, the device includes at least one processing unit 1302for executing instructions that can be stored in a memory device orelement 1304. As would be apparent to one of ordinary skill in the art,the device can include many types of memory, data storage, orcomputer-readable media, such as a first data storage for programinstructions for execution by the processing unit(s) 1302, the same orseparate storage can be used for images or data, a removable memory canbe available for sharing information with other devices, and any numberof communication approaches can be available for sharing with otherdevices.

The device typically will include some type of display element 1306,such as a touch screen, electronic ink (e-ink), organic light emittingdiode (OLED) or liquid crystal display (LCD), although devices such asportable media players might convey information via other means, such asthrough audio speakers.

As discussed, the device in many embodiments will include at least oneimaging element 1308, such as one or more cameras that are able tocapture images of the surrounding environment and that are able to imagea user, people, or objects in the vicinity of the device.

The image capture element can include any appropriate technology, suchas a CCD image capture element having a sufficient resolution, focalrange, and viewable area to capture an image of the user when the useris operating the device. Methods for capturing images using a cameraelement with a computing device are well known in the art and will notbe discussed herein in detail. It should be understood that imagecapture can be performed using a single image, multiple images, periodicimaging, continuous image capturing, image streaming, etc. Further, adevice can include the ability to start and/or stop image capture, suchas when receiving a command from a user, application, or other device.

The example computing device 1300 also includes at least one orientationdetermining element 1310 able to determine and/or detect orientationand/or movement of the device. Such an element can include, for example,an accelerometer or gyroscope operable to detect movement (e.g.,rotational movement, angular displacement, tilt, position, orientation,motion along a non-linear path, etc.) of the device 1300. An orientationdetermining element can also include an electronic or digital compass,which can indicate a direction (e.g., north or south) in which thedevice is determined to be pointing (e.g., with respect to a primaryaxis or other such aspect).

As discussed, the device in many embodiments will include at least apositioning element 1312 for determining a location of the device (orthe user of the device). A positioning element can include or comprise aGPS or similar location-determining elements operable to determinerelative coordinates for a position of the device. As mentioned above,positioning elements may include wireless access points, base stations,etc., that may either broadcast location information or enabletriangulation of signals to determine the location of the device. Otherpositioning elements may include QR codes, barcodes, RFID tags, NFCtags, etc., that enable the device to detect and receive locationinformation or identifiers that enable the device to obtain the locationinformation (e.g., by mapping the identifiers to a correspondinglocation). Various embodiments can include one or more such elements inany appropriate combination.

As mentioned above, some embodiments use the element(s) to track thelocation of a device. Upon determining an initial position of a device(e.g., using GPS), the device of some embodiments may keep track of thelocation of the device by using the element(s), or in some instances, byusing the orientation determining element(s) as mentioned above, or acombination thereof. As should be understood, the algorithms ormechanisms used for determining a position and/or orientation can dependat least in part upon the selection of elements available to the device.

The example device also includes one or more wireless components 1314operable to communicate with one or more electronic devices within acommunication range of the particular wireless channel. The wirelesschannel can be any appropriate channel used to enable devices tocommunicate wirelessly, such as Bluetooth, cellular, NFC, or Wi-Fichannels. It should be understood that the device can have one or moreconventional wired communications connections as known in the art.

The device also includes a power system 1316, such as may include abattery operable to be recharged through conventional plug-inapproaches, or through other approaches such as capacitive chargingthrough proximity with a power mat or other such device. Various otherelements and/or combinations are possible as well within the scope ofvarious embodiments.

In some embodiments the device can include at least one additional inputdevice 1318 able to receive conventional input from a user. Thisconventional input can include, for example, a push button, touch pad,touch screen, wheel, joystick, keyboard, mouse, keypad, or any othersuch device or element whereby a user can input a command to the device.These I/O devices could even be connected by a wireless infrared orBluetooth or other link as well in some embodiments. Some devices alsocan include a microphone or other audio capture element that acceptsvoice or other audio commands. For example, a device might not includeany buttons at all, but might be controlled only through a combinationof visual and audio commands, such that a user can control the devicewithout having to be in contact with the device.

In some embodiments, a device can include the ability to activate and/ordeactivate detection and/or command modes, such as when receiving acommand from a user or an application, or retrying to determine an audioinput or video input, etc. In some embodiments, a device can include aninfrared detector or motion sensor, for example, which can be used toactivate one or more detection modes. For example, a device might notattempt to detect or communicate with devices when there is not a userin the room. If an infrared detector (i.e., a detector with one-pixelresolution that detects changes in state) detects a user entering theroom, for example, the device can activate a detection or control modesuch that the device can be ready when needed by the user, but conservepower and resources when a user is not nearby.

A computing device, in accordance with various embodiments, may includea light-detecting element that is able to determine whether the deviceis exposed to ambient light or is in relative or complete darkness. Suchan element can be beneficial in a number of ways. In certainconventional devices, a light-detecting element is used to determinewhen a user is holding a cell phone up to the user's face (causing thelight-detecting element to be substantially shielded from the ambientlight), which can trigger an action such as the display element of thephone to temporarily shut off (since the user cannot see the displayelement while holding the device to the user's ear). The light-detectingelement could be used in conjunction with information from otherelements to adjust the functionality of the device. For example, if thedevice is unable to detect a user's view location and a user is notholding the device but the device is exposed to ambient light, thedevice might determine that it has likely been set down by the user andmight turn off the display element and disable certain functionality. Ifthe device is unable to detect a user's view location, a user is notholding the device and the device is further not exposed to ambientlight, the device might determine that the device has been placed in abag or other compartment that is likely inaccessible to the user andthus might turn off or disable additional features that might otherwisehave been available. In some embodiments, a user must either be lookingat the device, holding the device or have the device out in the light inorder to activate certain functionality of the device. In otherembodiments, the device may include a display element that can operatein different modes, such as reflective (for bright situations) andemissive (for dark situations). Based on the detected light, the devicemay change modes.

Using the microphone, the device can disable other features for reasonssubstantially unrelated to power savings. For example, the device canuse voice recognition to determine people near the device, such aschildren, and can disable or enable features, such as Internet access orparental controls, based thereon. Further, the device can analyzerecorded noise to attempt to determine an environment, such as whetherthe device is in a car or on a plane, and that determination can help todecide which features to enable/disable or which actions are taken basedupon other inputs. If voice recognition is used, words can be used asinput, either directly spoken to the device or indirectly as picked upthrough conversation. For example, if the device determines that it isin a car, facing the user and detects a word such as “hungry” or “eat,”then the device might turn on the display element and displayinformation for nearby restaurants, etc. A user can have the option ofturning off voice recording and conversation monitoring for privacy andother such purposes.

In some of the above examples, the actions taken by the device relate todeactivating certain functionality for purposes of reducing powerconsumption. It should be understood, however, that actions cancorrespond to other functions that can adjust similar and otherpotential issues with use of the device. For example, certain functions,such as requesting Web page content, searching for content on a harddrive and opening various applications, can take a certain amount oftime to complete. For devices with limited resources, or that have heavyusage, a number of such operations occurring at the same time can causethe device to slow down or even lock up, which can lead toinefficiencies, degrade the user experience and potentially use morepower.

In order to address at least some of these and other such issues,approaches in accordance with various embodiments can also utilizeinformation such as user gaze direction to activate resources that arelikely to be used in order to spread out the need for processingcapacity, memory space and other such resources.

In some embodiments, the device can have sufficient processingcapability, and the imaging element and associated analyticalalgorithm(s) may be sensitive enough to distinguish between the motionof the device, motion of a user's head, motion of the user's eyes andother such motions, based on the captured images alone. In otherembodiments, such as where it may be desirable for the process toutilize a fairly simple imaging element and analysis approach, it can bedesirable to include at least one orientation determining element thatis able to determine a current orientation of the device. In oneexample, the at least one orientation determining element is at leastone single- or multi-axis accelerometer that is able to detect factorssuch as three-dimensional position of the device and the magnitude anddirection of movement of the device, as well as vibration, shock, etc.Methods for using elements such as accelerometers to determineorientation or movement of a device are also known in the art and willnot be discussed herein in detail. Other elements for detectingorientation and/or movement can be used as well within the scope ofvarious embodiments for use as the orientation determining element. Whenthe input from an accelerometer or similar element is used along withthe input from the camera, the relative movement can be more accuratelyinterpreted, allowing for a more precise input and/or a less compleximage analysis algorithm.

When using an imaging element of the computing device to detect motionof the device and/or user, for example, the computing device can use thebackground in the images to determine movement. For example, if a userholds the device at a fixed orientation (e.g. distance, angle, etc.) tothe user and the user changes orientation to the surroundingenvironment, analyzing an image of the user alone will not result indetecting a change in an orientation of the device. Rather, in someembodiments, the computing device can still detect movement of thedevice by recognizing the changes in the background imagery behind theuser. So, for example, if an object (e.g., a window, picture, tree,bush, building, car, etc.) moves to the left or right in the image, thedevice can determine that the device has changed orientation, eventhough the orientation of the device with respect to the user has notchanged. In other embodiments, the device may detect that the user hasmoved with respect to the device and adjust accordingly. For example, ifthe user tilts their head to the left or right with respect to thedevice, the content rendered on the display element may likewise tilt tokeep the content in orientation with the user.

Various approaches can be utilized for locating one or more desiredfeatures of a user's face to determine various aspects useful fordetermining relative orientation. For example, an image can be analyzedto determine the approximate location and size of a user's head or face.FIG. 14( a) illustrates an example wherein the approximate position andarea of a user's head or face 1400 is determined and a virtual “box”1402 is placed around the face as an indication of position using one ofa plurality of image analysis algorithms for making such adetermination. Using one algorithm, a virtual “box” is placed around auser's face and the position and/or size of this box is continuallyupdated and monitored in order to monitor relative user position.Similar algorithms can also be used to determine an approximate locationand area 1404 of each of the user's eyes (or in some cases the eyes intandem). By determining the location of the user's eyes as well,advantages can be obtained as it can be more likely that the imagedetermined to be the user's head actually includes the user's head, andit can be determined that the user is facing the device. Further, therelative movement of the user's eyes can be easier to detect than theoverall movement of the user's head when performing motions such asnodding or shaking the head back and forth. Monitoring box size alsohelps to provide distance information as well as directionalinformation, which can be helpful when generating a three-dimensionalmodel for modifying image information based on relative user position.

Various other algorithms can be used to determine the location offeatures on a user's face. For example, FIG. 14( b) illustrates anexample wherein various features on a user's face are identified andassigned a point location 1406 in the image. The system thus can detectvarious aspects of user features and can determine more subtle changesin orientation. Such an approach provides advantages over the generalapproach of FIG. 14( a) in certain situations, as various other featurescan be determined, in case the user's eyes cannot be seen due toglasses, hair, etc.

Once the positions of facial features of a user are identified, relativemotion between the user and the device can be detected and utilized asinput. For example, FIG. 14( c) illustrates an example where the user'shead 1400 is moving up and down with respect to the viewable area of theimaging element. As discussed, this could be the result of the usermoving his or her head, or the user moving the device up and down, etc.FIG. 14( d) illustrates a similar example wherein the user is movingright to left relative to the device, through movement of the user, thedevice, or both. As can be seen, each movement can be tracked as avertical or horizontal movement, respectively, and each can be treateddifferently as an input to modify a displayed image. As should beunderstood, such a process also can detect diagonal or other suchmovements. FIG. 14( e) further illustrates an example wherein the usertilts the device and/or the user's head, and the relative change in eyeposition is detected as a rotation. In some systems, a “line” thatcorresponds to the relative position of the eyes can be monitored, and ashift in angle of this line can be compared to an angle threshold todetermine when the rotation should be interpreted as input. FIG. 14( f)illustrates another advantage of using an approach such as thatdescribed with respect to FIG. 14( b) to determine the position ofvarious features on a user's face. In this exaggerated example, it canbe seen that the features of a second user's head 1408 have a differentrelative position and separation. Thus, the device also can not onlydetermine positions of features for a user, but can distinguish betweendifferent users.

FIGS. 15( a) and 15(b) illustrate an example approach that can be usedto determine variations in relative distance between a user and a devicethat can be used in accordance with various embodiments. As in FIG. 15(a), the approximate position and area of a user's head or face 1500 isdetermined and a virtual “box” 1502 is placed around the face at aninitial distance as an indication of distance using one of a pluralityof image analysis algorithms for making such a determination. If theuser is known, the size of the user's head might be stored such that anactual distance to the user can be calculated based at least in partupon the size of the box 1502. If the user is not known, the distancecan be estimated or determined using other factors, such as stereoscopicimaging. In some embodiments, determinations will be relative withrespect to an initial box size when the actual distance cannot bedetermined

As the distance between the user and the device changes, the size of thevirtual box will change as well. For example, in FIG. 15( b) thedistance between the user and the device has increased, such that theuser's head 1520 appears smaller in the captured image information.Accordingly, the size of the virtual box 1522 for the adjusted size ofthe user's head is smaller than the original box 1502 for the initialdistance. By monitoring adjustments in the size of the box or anothermeasure of the user's head and/or other such features (e.g., boxes1524), the device can determine an approximate distance and/or change indistance to the user. As discussed, this information can be used toadjust aspects of the displayed image information such as a level ofzoom or amount of detail.

FIGS. 16( a) to 16(d) illustrate an example of how an interface plane orelement at different depths can be used to generate viewing-angleappropriate images in accordance with at least some embodiments. In FIG.16( a), the example orientation 1600 has a user 1602 substantially infront of a display element 1604 of a device. For simplicity ofexplanation, the interface plane or element here is represented in threedimensions, with a box 1606 on a background 1608. At the current viewingangle, the user is only able to see the top surface 1610 of theinterface plane or element 1606, as illustrated in the display view 1620of FIG. 16( b). In the orientation 1640 of FIG. 16( c), the device hasbeen rotated (or the user has moved with respect to the device). Toprovide an appropriate user experience in at least some embodiments, theinterface plane or element is effectively rotated with the device, suchthat the interface plane or element and background 1608 would rotateaccordingly. Based on the current viewing direction of the user 1602, itcan be seen in the display view 1660 of FIG. 16( d) that the viewableportion 1642 of the interface plane or element includes not only the topof the interface plane or element but at a level of depth (i.e., theinterface plane appears to be closer to a display screen of the device).By calculating this angle, the application can determine the portions ofthe top and side of the interface plane or element to display as aresult of the rotation. It also can be seen in FIG. 16( c) that any areaoccluded by the right side of the interface plane or element in FIG. 16(a) now can be seen, and that the area occluded by the left side of thebox is interface plane or element greater in FIG. 16( c).

In at least some embodiments, a computing device can utilize one or morecameras or other such sensors to determine the relative direction of theuser. For example, FIG. 17( a) illustrates an example situation 1700wherein a computing device 1702 is configured to utilize at least onecamera element 1706 to attempt to locate a feature of a user, such asthe user's head or eyes, for purposes of point of view determination. Inthis example, the user's eyes 1704 are located within the field of view1708 of a camera of the computing device 1702. As discussed elsewhereherein, however, the point of view of a user can be determined usingpositions of the user's eyes, pupils, head, or other such features thatcan be indicative of at least a general point of view. In someembodiments, the device might look for an object held by or otherwiseassociated with a user to determine a general point of view forrendering. Further, in some embodiments a device might utilize at leasttwo different cameras positioned on the device with a sufficientseparation such that the device can utilize stereoscopic imaging (oranther such approach) to determine a relative position of one or morefeatures, with respect to the device, in three dimensions. It should beunderstood that there can be additional imaging elements of the same ora different type at various other locations on the device as well withinthe scope of the various embodiments.

Software executing on the computing device (or otherwise incommunication with the computing device) can obtain information such asthe angular field of view of the camera, the zoom level at which theinformation is currently being captured, and any other such relevantinformation, which can enable the software to determine an approximatedirection 1710 of at least one of the user's eyes with respect to thecamera. In many embodiments, direction information will be sufficient toprovide adequate point-of-view dependent rendering. In at least someembodiments, however, it can also be desirable to determine distance tothe user in order to provide a more consistent and accurate rendering.In some embodiments, methods such as ultrasonic detection, feature sizeanalysis, luminance analysis through active illumination, or other suchdistance measurement approaches can be used to assist with positiondetermination. In other embodiments, a second camera can be used toenable distance determinations through stereoscopic imaging. Once thedirection vectors from at least two image capture elements aredetermined for a given feature, the intersection point of those vectorscan be determined, which corresponds to the approximate relativeposition in three dimensions of the respective feature as known fordisparity mapping and other such processes.

Further illustrating such an example approach, FIG. 17( b) illustratesan example image 1720 that could be captured of the user's head and eyesusing the camera 1706 of FIG. 17( a). One or more image analysisalgorithms can be used to analyze the image to perform patternrecognition, shape recognition, or another such process to identify afeature of interest, such as the user's eyes. Approaches to identifyinga feature in an image, such may include feature detection, facialfeature extraction, feature recognition, stereo vision sensing,character recognition, attribute estimation, or radial basis function(RBF) analysis approaches, are well known in the art and will not bediscussed herein in detail. As illustrated in this example, both eyes ofthe user might be able to be located in the captured image information.At least some algorithms are able to determine an approximate locationor region 1722, 1724 for each eye, or at least an approximate location1728 of the user's head, where at least one of those locations orregions is used for point of view determinations. Depending on factorssuch as the desired level of sensitivity and distance between the userand the device, however, such information can impact the accuracy of thepoint of view determinations. Approaches in accordance with variousembodiments can take advantage of the fact that the human brain combinesand processes information from both eyes to provide a “single” point ofview. Thus, the software can attempt to determine an intermediate point1726 between the user's eyes to use for the user's point of view.Various other approaches can be used as well, such as are discussedelsewhere herein. Once a relative location is determined in the imageinformation, the device can use information such as the field of view ofthe camera, the position of the camera with respect to the device, thezoom level of the camera, and other such information to determine arelative direction of the user, with that relative direction being usedfor the point of view to use in rendering the interface.

When using a camera to track location, however, the accuracy is limitedat least in part by the frame rate of the camera. Further, images takesome time to process such that there can be some lag in thedeterminations. As changes in orientation of the device can occurrelatively quickly, it can be desirable in at least some embodiments toenhance the accuracy of the point of view determinations. In someembodiments, a sensor or other such element of a computing device can beused to determine motions of the computing device, which can help adjustpoint of view determinations. The sensors can be any appropriate sensorscapable of providing information about rotations and/or translations ofthe device, as may include accelerometers, inertial sensors, electronicgyroscopes, electronic compasses, and the like.

For example, FIG. 18( a) illustrates a “top view” 1800 of a computingdevice 1802 operable to capture an image of an object 1804 (e.g., auser's head) within an angular view 1808 of a camera 1810 of thecomputing device. In this example, the computing device 1802 includes atleast one orientation- or rotation-determining element, such as anelectronic compass or electronic gyroscope, that is able to determine aframe of reference 1806 in two or three dimensions with respect to afirst orientation of the device. In at least some embodiments, anelectronic compass might be used to determine an axis of the frame ofreference 1806, as may correspond to a North direction, etc. In otherembodiments, a component such as an electronic gyroscope might becalibrated periodically with a component such as a compass, but mightinstead determine changes in orientation along three axes of rotationover time. Various other approaches to determining changes inorientation along one, two, or three axes of rotation can be used aswell within the scope of the various embodiments.

A first frame of reference 1806 or orientation can be determined at ornear the time of capture of a first image by a camera 1810 of thecomputing device 1802. In some embodiments, the determination can betriggered by receiving input to capture an image or another such action,but in other embodiments the frame of reference and/or orientationinformation can be updated periodically, such as several times a secondbased upon the type and/or configuration of the electronic gyroscope.The gyroscope can also be any appropriate electronic gyroscopecomponent, such as a conventional MEMS gyroscope used in variousconsumer devices. Approaches for implementing and obtaining orientationchanges from such a gyroscope are well known in the art and, as such,will not be discussed in detail herein.

FIG. 18( b) illustrates a second top view 1810 after a change inorientation of the computing device 1802. The electronic gyroscope (orother such component or embedded sensor) can detect the change inorientation, in this example corresponding to a change in angle 1812with respect to the frame of reference in the plane of the figure. Thegyroscope can present information about the change in orientation in anyappropriate form, such as in angles or radians of change for one, two,or three degrees (e.g., Δx, Δy, Δz), percentage changes in pitch, roll,and yaw, etc. In this example, the change in orientation is determinedto be a given angular amount of rotation 1812 about a single axis. Asillustrated, this causes the object 1804 to be moved to the right edgeof the field of view 1808 of the camera 1810. In at least someembodiments, the gyroscope may not be accurate enough to provide anexact amount of rotation, but can provide an approximation or estimateof the amount of rotation that can be used to narrow the search spaceand facilitate the location of corresponding objects in the images.Further, the information can provide a faster adjustment or predictionof relative position than can be provided from the camera in at leastsome embodiments. A similar approach can be used for translation,although the effects of translation on objects in captured images can bemuch less significant than angular changes, such that the imageinformation might be sufficient to account for translation changes in atleast some embodiments.

As discussed, different approaches can be implemented in variousenvironments in accordance with the described embodiments. For example,FIG. 19 illustrates an example of an environment 1900 for implementingaspects in accordance with various embodiments. As will be appreciated,although a Web-based environment is used for purposes of explanation,different environments may be used, as appropriate, to implement variousembodiments. The system includes electronic client devices 1918, 1920,1922, and 1924, which can include any appropriate device operable tosend and receive requests, messages or information over an appropriatenetwork 1904 and convey information back to a user of the device.Examples of such client devices include personal computers, cell phones,handheld messaging devices, laptop computers, set-top boxes, personaldata assistants, electronic book readers and the like. The network caninclude any appropriate network, including an intranet, the Internet, acellular network, a local area network or any other such network orcombination thereof. The network could be a “push” network, a “pull”network, or a combination thereof. In a “push” network, one or more ofthe servers push out data to the client device. In a “pull” network, oneor more of the servers send data to the client device upon request forthe data by the client device. Components used for such a system candepend at least in part upon the type of network and/or environmentselected. Protocols and components for communicating via such a networkare well known and will not be discussed herein in detail. Communicationover the network can be enabled via wired or wireless connections andcombinations thereof. In this example, the network includes theInternet, as the environment includes a Web server 1906 for receivingrequests and serving content in response thereto, although for othernetworks, an alternative device serving a similar purpose could be used,as would be apparent to one of ordinary skill in the art.

The illustrative environment includes at least one application server1908 and a data store 1910. It should be understood that there can beseveral application servers, layers or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. As used herein, the term “data store” refers to any deviceor combination of devices capable of storing, accessing and retrievingdata, which may include any combination and number of data servers,databases, data storage devices and data storage media, in any standard,distributed or clustered environment. The application server 1908 caninclude any appropriate hardware and software for integrating with thedata store 1910 as needed to execute aspects of one or more applicationsfor the client device and handling a majority of the data access andbusiness logic for an application. The application server providesaccess control services in cooperation with the data store and is ableto generate content such as text, graphics, audio and/or video to betransferred to the user, which may be served to the user by the Webserver 1906 in the form of HTML, XML or another appropriate structuredlanguage in this example. The handling of all requests and responses, aswell as the delivery of content between the client devices 1918, 1920,1922, and 1924 and the application server 1908, can be handled by theWeb server 1906. It should be understood that the Web and applicationservers are not required and are merely example components, asstructured code discussed herein can be executed on any appropriatedevice or host machine as discussed elsewhere herein.

The data store 1910 can include several separate data tables, databasesor other data storage mechanisms and media for storing data relating toa particular aspect. For example, the data store illustrated includesmechanisms for storing content (e.g., production data) 1912 and userinformation 1916, which can be used to serve content for the productionside. The data store is also shown to include a mechanism for storinglog or session data 1914. It should be understood that there can be manyother aspects that may need to be stored in the data store, such as pageimage information and access rights information, which can be stored inany of the above listed mechanisms as appropriate or in additionalmechanisms in the data store 1919. The data store 1919 is operable,through logic associated therewith, to receive instructions from theapplication server 1908 and obtain, update or otherwise process data inresponse thereto. In one example, a user might submit a search requestfor a certain type of item. In this case, the data store might accessthe user information to verify the identity of the user and can accessthe catalog detail information to obtain information about items of thattype. The information can then be returned to the user, such as in aresults listing on a Web page that the user is able to view via abrowser on anyone of the user devices 1918, 1920, 1922 and 1924.Information for a particular item of interest can be viewed in adedicated page or window of the browser.

Each server typically will include an operating system that providesexecutable program instructions for the general administration andoperation of that server and typically will include computer-readablemedium storing instructions that, when executed by a processor of theserver, allow the server to perform its intended functions. Suitableimplementations for the operating system and general functionality ofthe servers are known or commercially available and are readilyimplemented by persons having ordinary skill in the art, particularly inlight of the disclosure herein.

The environment in one embodiment is a distributed computing environmentutilizing several computer systems and components that areinterconnected via communication links, using one or more computernetworks or direct connections. However, it will be appreciated by thoseof ordinary skill in the art that such a system could operate equallywell in a system having fewer or a greater number of components than areillustrated in FIG. 19. Thus, the depiction of the system 1900 in FIG.19 should be taken as being illustrative in nature and not limiting tothe scope of the disclosure.

The various embodiments can be further implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers or computing devices which can be used to operate any of anumber of applications. User or client devices can include any of anumber of general purpose personal computers, such as desktop or laptopcomputers running a standard operating system, as well as cellular,wireless and handheld devices running mobile software and capable ofsupporting a number of networking and messaging protocols. Such a systemcan also include a number of workstations running any of a variety ofcommercially-available operating systems and other known applicationsfor purposes such as development and database management. These devicescan also include other electronic devices, such as dummy terminals,thin-clients, gaming systems and other devices capable of communicatingvia a network.

Most embodiments utilize at least one network that would be familiar tothose skilled in the art for supporting communications using any of avariety of commercially-available protocols, such as TCP/IP, OSI, FTP,UPnP, NFS, CIFS and AppleTalk. The network can be, for example, a localarea network, a wide-area network, a virtual private network, theInternet, an intranet, an extranet, a public switched telephone network,an infrared network, a wireless network and any combination thereof.

In embodiments utilizing a Web server, the Web server can run any of avariety of server or mid-tier applications, including HTTP servers, FTPservers, CGI servers, data servers, Java servers and businessapplication servers. The server(s) may also be capable of executingprograms or scripts in response requests from user devices, such as byexecuting one or more Web applications that may be implemented as one ormore scripts or programs written in any programming language, such asJava®, C, C# or C++ or any scripting language, such as Perl, Python orTCL, as well as combinations thereof. The server(s) may also includedatabase servers, including without limitation those commerciallyavailable from Oracle®, Microsoft®, Sybase® and IBM®.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (SAN) familiar to those skilled inthe art. Similarly, any necessary files for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (CPU), at least one inputdevice (e.g., a mouse, keyboard, controller, touch-sensitive displayelement or keypad) and at least one output device (e.g., a displaydevice, printer or speaker). Such a system may also include one or morestorage devices, such as disk drives, optical storage devices andsolid-state storage devices such as random access memory (RAM) orread-only memory (ROM), as well as removable media devices, memorycards, flash cards, etc.

Such devices can also include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device) and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium representing remote, local, fixed and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services or other elementslocated within at least one working memory device, including anoperating system and application programs such as a client applicationor Web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets) or both. Further, connection to other computing devices suchas network input/output devices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules or other data, including RAM, ROM, EEPROM, flash memoryor other memory technology, CD-ROM, digital versatile disk (DVD) orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices or any other medium which canbe used to store the desired information and which can be accessed by asystem device. Based on the disclosure and teachings provided herein, aperson of ordinary skill in the art will appreciate other ways and/ormethods to implement the various embodiments.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

What is claimed is:
 1. A computing device, comprising: a display screen;a camera; at least one computing device processor; a memory deviceincluding instructions that, when executed by the at least one computingdevice processor, enable the computing device to: associate a firstplane of content of a plurality of planes of content with a first nodeof a plurality of nodes of a node hierarchy and a second plane ofcontent of the plurality of planes of content with a second node of theplurality of nodes, the first plane of content including first contentand the second plane of content including second content, the firstcontent and the second content capable of being displayed on a displayscreen of a computing device; determine depth relationships among thefirst plane of content and the second plane of content based at least inpart upon a first depth position associated with the first plane ofcontent and a second depth position associated with the second plane ofcontent; determine a draw order in which to display the first plane ofcontent relative to the second plane of content, the draw order beingbased at least in part upon the depth relationships among the firstplane of content and the second plane of content, wherein planes ofcontent associated with nodes at a lowest depth position are displayedfirst; and display, on the display screen, the first content included inthe first plane of content and the second content included in the secondplane of content in an order based at least in part upon the draw order.2. The computing device of claim 1, further including a virtual lightsource, wherein the first depth position is a lower depth positionrelative to the display screen than the second depth position, andwherein the instructions, when executed, further cause the computingdevice to: render at least one shadow on the first plane of contentbased at least in part on an intersection of virtual light from thevirtual light source with the second plane of content.
 3. The computingdevice of claim 1, wherein the first depth position is a lower depthposition relative to the display screen than the second depth position,and wherein the instructions, when executed, further cause the computingdevice to: display the first content included in the first plane ofcontent before displaying the second content included in the secondplane of content.
 4. The computing device of claim 1, wherein each planeof content of the plurality of planes of content corresponds to a nodeof a plurality of nodes of a hierarchy, wherein the plurality of nodesof the hierarchy include at least one parent node and one or more childnodes depending from the at least one parent node, a depth position ofthe one or more child nodes being relative to at least one of thedisplay screen or the at least one parent node.
 5. A computerimplemented method, comprising: associating each one of a plurality ofplanes to a corresponding one of a subset of a plurality of nodes of anode hierarchy, each plane of the plurality of planes including contentcapable of being displayed on a display screen of a computing device;determining depth relationships among the plurality of planes based atleast upon position information included in the subset of the pluralityof nodes, each node of the subset of the plurality of nodes includingposition information corresponding to a display position relative to thedisplay screen for a corresponding plane of the plurality of planes;determining a draw order in which to render the plurality of planes, thedraw order being based at least in part upon the depth relationshipsamong the plurality of planes; and displaying, on the display screen, afirst view of the content included in each of the plurality of planes,each plane of the plurality of planes being displayed according to itsrespective position information and in an order corresponding to thedraw order, the first view being rendered from a perspectivecorresponding to a location of a viewer with respect to the displayscreen.
 6. The computer implemented method of claim 5, whereindetermining depth relationships among the plurality of planes includes:determining at least one of a depth position of one or more of theplurality of planes relative to the display screen or a depth positionof one or more of the plurality of planes relative to another plane. 7.The computer implemented method of claim 5, further comprising:rendering, based at least upon the draw order, a first plane of theplurality of planes with a first depth position before rendering asecond plane of the plurality of planes with a second depth position,the second plane being displayed to appear closer to a viewer of thecomputing device than first plane of the plurality of planes.
 8. Thecomputer implemented method of claim 7, wherein the first depth positionis a lower depth position relative to the display screen than the seconddepth position of the second plane, and wherein the method furtherincludes: rendering a shadow on the first plane based at least in parton an intersection of virtual light from a virtual light source with thesecond plane.
 9. The computer implemented method of claim 5, furthercomprising: determining a change in position of first content in a firstplane of the plurality of planes, wherein the change in position iscaused by at least one of an animation of the first content or ascrolling of an interface displaying the first content; updatingposition information for the first plane based at least upon the changein position; and updating position information for at least one otherplane of the plurality of planes based at least upon the change inposition of the first content.
 10. The computer implemented method ofclaim 5, wherein displaying the first view further includes: displayingat least one plane of the plurality of planes to appear to be closer toa viewer of the computing device than at least one other plane of theplurality of planes.
 11. The computer implemented method of claim 5,wherein displaying the first view further includes: adjusting anappearance of at least one of the plurality of planes according to thelocation of the viewer, wherein adjusting the appearance of the at leastone of the plurality of planes includes adjusting at least one of asize, shape, color, shading, or blur of the at least one of theplurality of planes according to the location of the viewer, and whereincontent included in the at least one of the plurality of planes inheritsat least some adjustments to the appearance of the at least one of theplurality of planes.
 12. The computer implemented method of claim 5,further comprising: determining a change in orientation of the computingdevice relative to the viewer of the display screen; determining asecond location of the viewer with respect to the computing device, thesecond location being determined based on the change in the orientation;and displaying, on the display screen, a second view of the contentbased at least in part on a lateral position and a depth positionassociated with each of the plurality of planes, the second view beingrendered from a perspective of the second location.
 13. The computerimplemented method of claim 5, further comprising: assigning a firstplane of the plurality of planes to a first depth position; assigning asecond plane of the plurality of planes to a second depth position, thefirst depth position being different from the second depth position;determining a change in position of the computing device relative to theviewer of the display screen; determining a second location of theviewer with respect to the computing device, the second location of theviewer corresponding to the change in position; and rendering a changefrom a first lateral position to a second lateral position for each ofthe first plane and the second plane based at least in part upon thefirst depth position for the first plane, the second depth position forthe second plane, and the second location of the viewer relative to thecomputing device.
 14. The computer implemented method of claim 5,further comprising: determining a change in position of the computingdevice relative to the viewer of the display screen; determining asecond location of the viewer with respect to the computing device, thesecond location of the viewer corresponding to the change in position;shifting a position of a first plane of the plurality of planes based atleast upon the second location of the viewer, the shifting providing adisplay of a second plane of the plurality of planes previously occludedby the first plane of the plurality of planes.
 15. The computerimplemented method of claim 5, further comprising: determining, based atleast in part upon the location of the viewer, coefficients for a set oftransformation equations to adjust a scale and a translation for arespective plane with respect to a determined pivot point associatedwith a center of the display screen; and adjusting an appearance ofcontent associated with at least one of the plurality of planes usingthe set of transformation equations.
 16. The computer implemented methodof claim 5, wherein a first element in a first plane of the plurality ofplanes is capable of partially obscuring a second element in a secondplane of the plurality of planes based at least in part upon the depthrelationships.
 17. A non-transitory computer readable storage mediumstoring one or more sequences of instructions executable by one or moreprocessors to perform a set of operations comprising: associating eachone of a plurality of planes to a corresponding one of a subset of aplurality of nodes of a node hierarchy, each plane of the plurality ofplanes including content capable of being displayed on a display screenof a computing device; determining depth relationships among theplurality of planes based at least upon position information included inthe subset of the plurality of nodes, each node of the subset of theplurality of nodes including position information corresponding to adisplay position relative to the display screen for a correspondingplane of the plurality of planes; determining a draw order in which torender the plurality of planes, the draw order being based at least inpart upon the depth relationships among the plurality of planes; anddisplaying, on the display screen, a first view of the content includedin each one of the plurality of planes, each plane of the plurality ofplanes being displayed according to respective position information andin an order corresponding to the draw order, the first view beingrendered from a perspective corresponding to a location of a viewer withrespect to the display screen.
 18. The non-transitory computer readablestorage medium of claim 17, further comprising instructions executed bythe one or more processors to perform the operations of: rendering,based at least upon the draw order, a first plane of the plurality ofplanes with a first depth position before rendering a second plane ofthe plurality of planes with a second depth position, the first planebeing displayed to appear closer to a viewer of the computing devicethan second plane of the plurality of planes.
 19. The non-transitorycomputer readable storage medium of claim 17, further comprisinginstructions executed by the one or more processors to perform theoperations of: determining a change in position of first content in afirst plane of the plurality of planes, wherein the change in positionis caused by at least one of an animation of the first content or ascrolling of an interface displaying the first content; updatingposition information for the first plane based at least upon the changein position; and updating position information for at least one otherplane of the plurality of planes based at least upon the change inposition of the first content.
 20. The non-transitory computer readablestorage medium of claim 17, further comprising instructions executed bythe one or more processors to perform the operations of: displaying atleast one plane of the plurality of planes to appear to be closer to aviewer of the computing device than at least one other plane of theplurality of planes.